Reinforcement Learning Algorithm for Partially Observable Markov Decision Problems

Increasing attention has been paid to reinforcement learning algo rithms in recent years partly due to successes in the theoretical analysis of their behavior in Markov environments If the Markov assumption is removed however neither generally the algorithms nor the analyses continue to be usable We propose and analyze a new learning algorithm to solve a certain class of non Markov decision problems Our algorithm applies to problems in which the environment is Markov but the learner has restricted access to state information The algorithm involves a Monte Carlo pol icy evaluation combined with a policy improvement method that is similar to that of Markov decision problems and is guaranteed to converge to a local maximum The algorithm operates in the space of stochastic policies a space which can yield a policy that per forms considerably better than any deterministic policy Although the space of stochastic policies is continuous even for a discrete action space our algorithm is computationally tractable